{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.<br>\n",
    "Licensed under the MIT License.</i>\n",
    "<br>\n",
    "# Model Comparison for NCF Using the Neural Network Intelligence Toolkit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows how to use the **[Neural Network Intelligence](https://nni.readthedocs.io/en/latest/) toolkit (NNI)** for tuning hyperparameters for the Neural Collaborative Filtering Model.\n",
    "\n",
    "To learn about each tuner NNI offers you can read about it [here](https://nni.readthedocs.io/en/latest/Tuner/BuiltinTuner.html).\n",
    "\n",
    "NNI is a toolkit to help users design and tune machine learning models (e.g., hyperparameters), neural network architectures, or complex system’s parameters, in an efficient and automatic way. NNI has several appealing properties: ease of use, scalability, flexibility and efficiency. NNI can be executed in a distributed way on a local machine, a remote server, or a large scale training platform such as OpenPAI or Kubernetes. \n",
    "\n",
    "In this notebook, we can see how NNI works with two different model types and the differences between their hyperparameter search spaces, yaml config file, and training scripts.\n",
    "\n",
    "- [NCF Training Script](../../reco_utils/nni/ncf_training.py)\n",
    "\n",
    "For this notebook we use a _local machine_ as the training platform (this can be any machine running the `reco_base` conda environment). In this case, NNI uses the available processors of the machine to parallelize the trials, subject to the value of `trialConcurrency` we specify in the configuration. Our runs and the results we report were obtained on a [Standard_D16_v3 virtual machine](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dv3-series-1) with 16 vcpus and 64 GB memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Global Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 23:51:54) \n",
      "[GCC 7.3.0]\n",
      "Tensorflow version: 1.15.2\n",
      "NNI version: 1.5\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "import json\n",
    "import os\n",
    "import surprise\n",
    "import papermill as pm\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import subprocess\n",
    "import tensorflow as tf\n",
    "import yaml\n",
    "import pkg_resources\n",
    "from tempfile import TemporaryDirectory\n",
    "\n",
    "import reco_utils\n",
    "from reco_utils.common.timer import Timer\n",
    "from reco_utils.dataset import movielens\n",
    "from reco_utils.dataset.python_splitters import python_chrono_split\n",
    "from reco_utils.evaluation.python_evaluation import rmse, precision_at_k, ndcg_at_k\n",
    "from reco_utils.tuning.nni.nni_utils import (\n",
    "    check_experiment_status, \n",
    "    check_stopped, \n",
    "    check_metrics_written, \n",
    "    get_trials,\n",
    "    stop_nni, start_nni\n",
    ")\n",
    "from reco_utils.recommender.ncf.dataset import Dataset as NCFDataset\n",
    "from reco_utils.recommender.ncf.ncf_singlenode import NCF\n",
    "from reco_utils.tuning.nni.ncf_utils import compute_test_results, combine_metrics_dicts\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Tensorflow version: {}\".format(tf.__version__))\n",
    "print(\"NNI version: {}\".format(pkg_resources.get_distribution(\"nni\").version))\n",
    "\n",
    "tmp_dir = TemporaryDirectory()\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Prepare Dataset\n",
    "1. Download data and split into training, validation and test sets\n",
    "2. Store the data sets to a local directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters used by papermill\n",
    "# Select Movielens data size: 100k, 1m\n",
    "MOVIELENS_DATA_SIZE = '100k'\n",
    "SURPRISE_READER = 'ml-100k'\n",
    "TMP_DIR = tmp_dir.name\n",
    "NUM_EPOCHS = 10\n",
    "MAX_TRIAL_NUM = 16\n",
    "DEFAULT_SEED = 42\n",
    "\n",
    "# time (in seconds) to wait for each tuning experiment to complete\n",
    "WAITING_TIME = 20\n",
    "MAX_RETRIES = MAX_TRIAL_NUM*4 # it is recommended to have MAX_RETRIES>=4*MAX_TRIAL_NUM\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4.81k/4.81k [00:00<00:00, 8.54kKB/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>itemID</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3.0</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3.0</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1.0</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2.0</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1.0</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  itemID  rating  timestamp\n",
       "0     196     242     3.0  881250949\n",
       "1     186     302     3.0  891717742\n",
       "2      22     377     1.0  878887116\n",
       "3     244      51     2.0  880606923\n",
       "4     166     346     1.0  886397596"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note: The NCF model can incorporate\n",
    "df = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=[\"userID\", \"itemID\", \"rating\", \"timestamp\"]\n",
    ")\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, validation, test = python_chrono_split(df, [0.7, 0.15, 0.15])\n",
    "train = train.drop(['timestamp'], axis=1)\n",
    "validation = validation.drop(['timestamp'], axis=1)\n",
    "test = test.drop(['timestamp'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "LOG_DIR = os.path.join(TMP_DIR, \"experiments\")\n",
    "os.makedirs(LOG_DIR, exist_ok=True)\n",
    "\n",
    "DATA_DIR = os.path.join(TMP_DIR, \"data\") \n",
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "\n",
    "TRAIN_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_train.pkl\"\n",
    "train.to_pickle(os.path.join(DATA_DIR, TRAIN_FILE_NAME))\n",
    "\n",
    "VAL_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_val.pkl\"\n",
    "validation.to_pickle(os.path.join(DATA_DIR, VAL_FILE_NAME))\n",
    "\n",
    "TEST_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_test.pkl\"\n",
    "test.to_pickle(os.path.join(DATA_DIR, TEST_FILE_NAME))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Prepare Hyperparameter Tuning "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run an experiment on NNI we require a general training script for our model of choice.\n",
    "A general framework for a training script utilizes the following components\n",
    "1. Argument Parse for the fixed parameters (dataset location, metrics to use)\n",
    "2. Data preprocessing steps specific to the model\n",
    "3. Fitting the model on the train set\n",
    "4. Evaluating the model on the validation set on each metric (ranking and rating)\n",
    "5. Save metrics and model\n",
    "\n",
    "To utilize NNI we also require a hypeyparameter search space. Only the hyperparameters we want to tune are required in the dictionary. NNI supports different methods of [hyperparameter sampling](https://nni.readthedocs.io/en/latest/Tutorial/SearchSpaceSpec.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `script_params` below are the parameters of the training script that are fixed (unlike `hyper_params` which are tuned)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "PRIMARY_METRIC = \"precision_at_k\"\n",
    "RATING_METRICS = [\"rmse\"]\n",
    "RANKING_METRICS = [\"precision_at_k\", \"ndcg_at_k\"]  \n",
    "USERCOL = \"userID\"\n",
    "ITEMCOL = \"itemID\"\n",
    "REMOVE_SEEN = True\n",
    "K = 10\n",
    "RANDOM_STATE = 42\n",
    "VERBOSE = True\n",
    "BIASED = True\n",
    "\n",
    "script_params = \" \".join([\n",
    "    \"--datastore\", DATA_DIR,\n",
    "    \"--train-datapath\", TRAIN_FILE_NAME,\n",
    "    \"--validation-datapath\", VAL_FILE_NAME,\n",
    "    \"--surprise-reader\", SURPRISE_READER,\n",
    "    \"--rating-metrics\", \" \".join(RATING_METRICS),\n",
    "    \"--ranking-metrics\", \" \".join(RANKING_METRICS),\n",
    "    \"--usercol\", USERCOL,\n",
    "    \"--itemcol\", ITEMCOL,\n",
    "    \"--k\", str(K),\n",
    "    \"--random-state\", str(RANDOM_STATE),\n",
    "    \"--epochs\", str(NUM_EPOCHS),\n",
    "    \"--primary-metric\", PRIMARY_METRIC\n",
    "])\n",
    "\n",
    "if BIASED:\n",
    "    script_params += \" --biased\"\n",
    "if VERBOSE:\n",
    "    script_params += \" --verbose\"\n",
    "if REMOVE_SEEN:\n",
    "    script_params += \" --remove-seen\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We specify the search space for the NCF hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ncf_hyper_params = {\n",
    "    'n_factors': {\"_type\": \"choice\", \"_value\": [2, 4, 8, 12]},\n",
    "    'learning_rate': {\"_type\": \"uniform\", \"_value\": [1e-3, 1e-2]},\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(TMP_DIR, 'search_space_ncf.json'), 'w') as fp:\n",
    "    json.dump(ncf_hyper_params, fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This config file follows the guidelines provided in [NNI Experiment Config instructions](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/ExperimentConfig.md).\n",
    "\n",
    "The options to pay attention to are\n",
    "- The \"searchSpacePath\" which contains the space of hyperparameters we wanted to tune defined above\n",
    "- The \"tuner\" which specifies the hyperparameter tuning algorithm that will sample from our search space and optimize our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"authorName\": \"default\",\n",
    "    \"experimentName\": \"tensorflow_ncf\",\n",
    "    \"trialConcurrency\": 8,\n",
    "    \"maxExecDuration\": \"1h\",\n",
    "    \"maxTrialNum\": MAX_TRIAL_NUM,\n",
    "    \"trainingServicePlatform\": \"local\",\n",
    "    # The path to Search Space\n",
    "    \"searchSpacePath\": \"search_space_ncf.json\",\n",
    "    \"useAnnotation\": False,\n",
    "    \"logDir\": LOG_DIR,\n",
    "    \"tuner\": {\n",
    "        \"builtinTunerName\": \"TPE\",\n",
    "        \"classArgs\": {\n",
    "            #choice: maximize, minimize\n",
    "            \"optimize_mode\": \"maximize\"\n",
    "        }\n",
    "    },\n",
    "    # The path and the running command of trial\n",
    "    \"trial\":  {\n",
    "      \"command\": f\"{sys.executable} ncf_training.py {script_params}\",\n",
    "      \"codeDir\": os.path.join(os.path.split(os.path.abspath(reco_utils.__file__))[0], \"tuning\", \"nni\"),\n",
    "      \"gpuNum\": 0\n",
    "    }\n",
    "}\n",
    " \n",
    "with open(os.path.join(TMP_DIR, \"config_ncf.yml\"), \"w\") as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Execute NNI Trials\n",
    "\n",
    "The conda environment comes with NNI installed, which includes the command line tool `nnictl` for controlling and getting information about NNI experiments. <br>\n",
    "To start the NNI tuning trials from the command line, execute the following command: <br>\n",
    "`nnictl create --config <path of config.yml>` <br>\n",
    "\n",
    "\n",
    "The `start_nni` function will run the `nnictl create` command. To find the URL for an active experiment you can run `nnictl webui url` on your terminal.\n",
    "\n",
    "In this notebook the 16 NCF models are trained concurrently in a single experiment with batches of 8. While NNI can run two separate experiments simultaneously by adding the `--port <port_num>` flag to `nnictl create`, the total training time will probably be the same as running the batches sequentially since these are CPU bound processes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_nni()\n",
    "config_path_ncf = os.path.join(TMP_DIR, 'config_ncf.yml')\n",
    "with Timer() as time_ncf:\n",
    "    start_nni(config_path_ncf, wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_metrics_written(wait=WAITING_TIME, max_retries=MAX_RETRIES)\n",
    "trials_ncf, best_metrics_ncf, best_params_ncf, best_trial_path_ncf = get_trials('maximize')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'rmse': 3.2212178182820117,\n",
       " 'ndcg_at_k': 0.1439899349063176,\n",
       " 'precision_at_k': 0.11633085896076353}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_metrics_ncf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'parameter_id': 3,\n",
       " 'parameter_source': 'algorithm',\n",
       " 'parameters': {'n_factors': 12, 'learning_rate': 0.0023365527461525885},\n",
       " 'parameter_index': 0}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params_ncf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Baseline Model\n",
    "\n",
    "Although we hope that the additional effort of utilizing an AutoML framework like NNI for hyperparameter tuning will lead to better results, we should also draw comparisons using our baseline model (our model trained with its default hyperparameters). This allows us to precisely understand what performance benefits NNI is or isn't providing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = NCFDataset(train, validation, seed=DEFAULT_SEED)\n",
    "model = NCF(\n",
    "    n_users=data.n_users, \n",
    "    n_items=data.n_items,\n",
    "    model_type=\"NeuMF\",\n",
    "    n_factors=4,\n",
    "    layer_sizes=[16,8,4],\n",
    "    n_epochs=NUM_EPOCHS,\n",
    "    learning_rate=1e-3,  \n",
    "    verbose=True,\n",
    "    seed=DEFAULT_SEED\n",
    ")\n",
    "model.fit(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'rmse': 3.2096711022473214,\n",
       " 'precision_at_k': 0.11145281018027572,\n",
       " 'ndcg_at_k': 0.13550842348404918}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results = compute_test_results(model, train, validation, RATING_METRICS, RANKING_METRICS)\n",
    "test_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Show Results\n",
    "\n",
    "The metrics for each model type is reported on the validation set. At this point we can compare the metrics for each model and select the one with the best score on the primary metric(s) of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rmse</th>\n",
       "      <th>precision_at_k</th>\n",
       "      <th>ndcg_at_k</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.209671</td>\n",
       "      <td>0.111453</td>\n",
       "      <td>0.135508</td>\n",
       "      <td>ncf_baseline</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.221218</td>\n",
       "      <td>0.116331</td>\n",
       "      <td>0.143990</td>\n",
       "      <td>ncf_tuned</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       rmse  precision_at_k  ndcg_at_k          name\n",
       "0  3.209671        0.111453   0.135508  ncf_baseline\n",
       "0  3.221218        0.116331   0.143990     ncf_tuned"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results['name'] = 'ncf_baseline'\n",
    "best_metrics_ncf['name'] = 'ncf_tuned'\n",
    "combine_metrics_dicts(test_results, best_metrics_ncf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the above metrics, we determine that NNI has identified a set of hyperparameters that does demonstrate an improvement on our metrics of interest. In this example, it turned out that an `n_factors` of 12 contributed to a better performance than an `n_factors` of 4. While the difference in `precision_at_k` and `ndcg_at_k` is small, NNI has helped us determine that a slightly larger embedding dimension for NCF may be useful for the movielens dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stop the NNI experiment \n",
    "stop_nni()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_dir.cleanup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Concluding Remarks\n",
    "\n",
    "In this notebook we showed how to use the NNI framework on different models. By inspection of the training scripts, the differences between the two should help you identify what components would need to be modified to run another model with NNI.\n",
    "\n",
    "In practice, an AutoML framework like NNI is just a tool to help you explore a large space of hyperparameters quickly with a pre-described level of randomization. It is recommended that in addition to using NNI one trains baseline models using typical hyperparamter choices (learning rate of 0.005, 0.001 or regularization rates of 0.05, 0.01, etc.) to draw  more meaningful comparisons between model performances. This may help determine if a model is overfitting from the tuner or if there is a statistically significant improvement.\n",
    "\n",
    "Another thing to note is the added computational cost required to train models using an AutoML framework. In this case, it takes about 6 minutes to train each of the models on a [Standard_NC6 VM](https://docs.microsoft.com/en-us/azure/virtual-machines/nc-series). With this in mind, while NNI can easily train hundreds of models over all hyperparameters for a model, in practice it may be beneficial to choose a subset of the hyperparameters that are deemed most important and to tune those. Too small of a hyperparameter search space may restrict our exploration, but too large may also lead to random noise in the data being exploited by a specific combination of hyperparameters.   \n",
    "\n",
    "For examples of scaling larger tuning workloads on clusters of machines, see [the notebooks](./README.md) that employ the [Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. References\n",
    "\n",
    "Recommenders Repo References\n",
    "* [NCF deep-dive notebook](../02_model/ncf_deep_dive.ipynb)\n",
    "* [SVD NNI notebook (uses more tuners available)](./nni_surprise_svd.ipynb)\n",
    "\n",
    "External References\n",
    "* [NCF Paper](https://arxiv.org/abs/1708.05031) \n",
    "* [NNI Docs | Neural Network Intelligence toolkit](https://github.com/Microsoft/nni)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_gpu)",
   "language": "python",
   "name": "reco_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}